The present invention relates to encryption and confidentiality of data on an external data storage system. More specifically, the invention relates to data reduction, including compression and de-duplication associated with storing encrypted data on the external storage system.
End-to-end encryption is the process of encrypting data close to the source before committing the encrypted data to storage. This encryption process has become increasingly prevalent due to security concerns regarding third party storage or cloud providers, domain-specific regulations mandating the encryption of sensitive data, ensuring secure deletion of data, and encryption requirements in high-security data centers. The client is the only entity in control of keys used to encrypt the data. Accordingly, no information is revealed to the cloud provider or other cloud provider tenants.
Encrypting data is limiting however, in that the majority of storage efficiency functions do not achieve their intended functions when operating on encrypted data. Encrypting data maximizes the entropy of ciphertext. As a consequence, encrypted data cannot be compressed. Furthermore, encryption of the same content in two different files or two different locations results in different ciphertexts, resulting in the failure of standard deduplication attempts.